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STEM  Degrees:  2  STEM  Participants:  10 

Major  Goals:  This  proposal  is  about  enabling  methods  to  better  understand  visual  perception  as  it  pertains  to 

dynamic  camouflage.  Cephalopod's  (e.g.,  octopuses  and  cuttlefish)  make  themselves  invisible  to 

extremely  sophisticated  visual  predation  using  dynamic  camouflage.  Despite  their  remarkable 

abilities,  these  animals  use  only  three  types  of  coloration  patterns:  Uniform,  Mottled,  and 

Disruptive.  Overlaid  on  this  color  camouflage  are  many  other  disguise  techniques:  the  cuttlefish, 

for  example,  also  uses  pattern  generation,  skin  texture,  and  body  posture  to  disguise  or  reveal 

themselves.  Though  some  basic  explanations  of  how  these  animals  perform  these  maneuvers  and 

when  they  choose  to  implement  them  has  recently  been  explained,  true  understanding  of  how  to 

control  and  use  dynamic  camouflage  in  a  human  context  will  only  be  revealed  using  synthetic 

systems,  instead  of  animal  models.  The  goal  of  this  proposal  is  to  develop  synthetic 

chromatophores  that  display  color  electrically.  To  create  stretchable  synthetic  chromatophores,  we  synthesized 
high  density  arrays  of  individually  addressable  elastomeric  light  emitting  capacitors  and  used  a  passive  matrix 
addressing  method.  In  this  report,  we  cover  the  results  of  two  proposal  periods:  September  201 5  -  August  2016 
and  September  2016  -  July  2017. 

Accomplishments:  o  We  reported  the  first  article  on  hyperelastic  light  emitting  capacitors  (HLEC),  resulting  in  a 
display  with  the  largest  reported  strain  to  failure  (-500%  linear  strain).  The  HLEC  may  actually  fail  at  higher  strains, 
the  display  stopped  working  at  500%  strain  due  to  experimental  limitations.  (Fig.  1-3). 

o  We  reported  the  first  color  changing  soft  robot  that  is  electrically  controlled.  This  soft  robot  is  composed  of 
silicone  elastomer  and  the  display  is  laminated  onto  its  skin,  which  deforms  by  100's%  strain.  (Fig.  4). 

o  We  used  the  HLECs  to  not  only  display  color  on  the  soft  robots,  but  also  sense  touch  and  for  feedback  control 
of  the  crawling  soft  robot.  (Fig.  4). 

o  We  developed  a  photolithography  and  transfer  printing  method  to  scale  up  the  pixel  density  of  the  HLEC 
display  and  build  a  passive  matrix  controller  to  display  individual  pixel  light  emission.  (Fig.  5-7). 

o  The  high  density  HLEC  array  also  serves  as  a  multi-point  touch  sensor.  (Fig.  8). 

o  While  discussing  collaboration  with  Dr.  Roger  Hanlon,  it  became  evident  that  the  ability  to  control  the 
polarization  of  reflected  light  is  important  to  study  cuttlefish  vision  and,  commensurately,  camouflage.  In  order  to 
control  the  polarization  of  our  HLECs,  it  would  require  a  stretchable  polarizer  layered  onto  the  system;  an  added 
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complexity  in  fabrication. 

o  We  reported  a  paper  on  Untethered  Stretchable  Displays  for  Haptic  Interaction.  This  display  is  powered  by 
batteries,  changes  shape  from  a  flat  disk  to  spherical  orb,  and  senses  touch  while  emitting  light.  (Fig.  9-14). 

o  We  reported  the  first  use  of  machine  learning  to  interpret  mechanical  deformation  as  gestures.  We  pressurized 
a  5x5  grid  of  stretchable  capacitors  into  a  hemisphere  and  used  it  as  a  haptic  interface.  By  classifying  different  the 
hemisphere  deformations,  we  were  able  to  train  gestures  to  control  a  game  of  Tetris.  (Fig.  15-18). 

o  Two  students  (Bryan  Peele  &  Chris  Larson)  successfully  defended  their  theses  and  graduated. 

Training  Opportunities:  Several  students  received  training  through  this  program: 

"  Chris  Larson  (PhD  candidate) 

"  Bryan  Peele  (PhD  candidate) 

"  Shuo  Li  (PhD  candidate) 

"  Jason  Cortell  (Staff) 

Of  these,  Chris  Larson  and  Bryan  Peele  have  received  their  PhD  and  Shuo  Li  is  close. 

Results  Dissemination:  "  Larson  C,  Peele  B,  Li  S,  Robinson  S,  Totaro  M,  Beccai  L,  Mazzolai  B,  Shepherd  R. 
Highly  stretchable  electroluminescent  skin  for  optical  signaling  and  tactile  sensing.  Science  (2016). 

"  Li  S,  Peele  B,  Larson  C,  Zhao  H,  Shepherd  R.  A  stretchable  multicolor  display  and  touch  interface  using 
photopatterning  and  transfer  printing.  Adv.  Mat.  (2017). 

"  Peele  B,  Li  S,  Larson  C,  Cortell  J,  Habtour  H,  Shepherd  R.  Untethered  stretchable  displays  for  tactile 
interaction.  Soft  Robotics  (accepted). 

"  Larson  C,  Spjut  J,  Knepper  R,  Shepherd  R.  A  Deformable  Interface  that  Learns  Touch  Gestures  using  Deep 
Neural  Networks.  Published  on  arXiv  (https://arxiv.org/abs/1706.02542). 

"  Li  S,  Zhao  H,  Shepherd  R.  Flexible  and  stretchable  sensors  for  fluidic  elastomer  actuated  soft  robots.  MRS 
Bulletin  (2017). 


Honors  and  Awards:  "PI  Shepherd  received  ONR  Young  Investigator  Award  during  this  time  period. 

"  Publication  in  Science  magazine 

Protocol  Activity  Status: 

Technoiogy  Transfer:  o  Provisional  patent  applications: 

o  62/250,172:  Stretchable  Electroluminescent  Devices  and  Methods  Making  and  Using  Same;  Photolithography 
procedure  for  mm2  area  HLEC  pixels  patterned  into  RGB  arrangements  using  transfer  printing 
o  62/317,834:  Ultra-Stretchable  Electroluminescent  Device;  Ionic  hydrogel  capacitors  with  embedded  light 
emitting  semiconducting  particles  in  the  dielectric  layer. 
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stretchable  Capacitors  That  Electrically  Luminesce,  Sense,  and 
Actuate  for  Biomimetic  Coloration,  Shepherd,  Cornell  University 


Objective:  Develop  a  platform  to  accurately  mimic 
cephalopod  skin  for  dynamic  coloration  and 
posture  to  study  visual  perception  in  cuttlefish. 

Scientific  Challenges. 

•Can  we  pattern  high  resolution  light  emitting 
dielectric  elastomer  actuators  (which  we  call 
HLECs). 

•  Can  we  control  them  effectively? 

•  Can  we  use  their  sensing  capabilities  for  shape 
control  of  soft  robots  and  haptic  interfaces? 

Major  Accomplishments: 


Potential  Army  Relevance: 


•Photolithography  procedure  for  high  density  RGB  HLEC 
pixels.  Li  et  al.,  Adv.  Mat.  (2016) 

•Applied  HLECs  to  posture  changing  soft  robot  for  dynamic 
coloration.  Larson  et  al.,  Science  (2016). 

•Portable  system  for  use  in  experiment  with  cuttlefish.  Peele 
et  al.,  SoRo  (accepted) 

•Review  on  stretchable  sensors.  Li  et  al.,  MRS  Bulletin 
(2017) 


•Stretchable  displays  for  visually  indicating 
locations  for  transport,  targets.  A  small  display  can 
turn  into  a  large  one  for  portability. 

•Wearable  dynamic  coloration  via  counter 
illumination  for  daytime  camouflage,  (may  require 
significant  further  development). 

•Stretchable  camouflage  netting  for  equipment. 


•Machine  learning  interpretation  of  touch  in  high  density 
stretchable  sensor  networks.  Larson  et  al.  IJRR  (submitted) 


•Damage  sensing  via  passive  matrix  sensor  array. 


•Stretchable  Transducers  for  Kinesthetic  Interactions  in 
Virtual  Reality  (SIGGRAPH  conference  presentation; 
collaboration  with  NVIDIA  Research). 


•Haptic  interfaces  &  displays  for  vehicles 
R  Shepherd:  rfs247@cornell.edu:  607  255  8654 


•  Project  Summary  -  Grant  #  W91 1NF-12-R-001 1-2 

(Final  Report:  September  2015  -  July  2017) 

Stretchable  capacitors  that  electrically  luminesce,  sense,  and  actuate  for  biomimetic  coloration 


Robert  Shepherd 

Mechanical  &  Aerospace  Engineering 
Cornell  University,  Ithaca,  NY,  14853 


Objective 

This  proposal  is  about  enabling  methods  to  better  understand  visual  perception  as  it  pertains  to 
dynamic  camouflage.  Cephalopod’s  (e.g.,  octopuses  and  cuttlefish)  make  themselves  invisible  to 
extremely  sophisticated  visual  predation  using  dynamic  camouflage.  Despite  their  remarkable 
abilities,  these  animals  use  only  three  types  of  coloration  patterns:  Uniform,  Mottled,  and 
Disruptive.  Overlaid  on  this  color  camouflage  are  many  other  disguise  techniques;  the  cuttlefish, 
for  example,  also  uses  pattern  generation,  skin  texture,  and  body  posture  to  disguise  or  reveal 
themselves.  Though  some  basic  explanations  of  how  these  animals  perform  these  maneuvers  and 
when  they  choose  to  implement  them  has  recently  been  explained,  true  understanding  of  how  to 
control  and  use  dynamic  camouflage  in  a  human  context  will  only  be  revealed  using  synthetic 
systems,  instead  of  animal  models.  The  goal  of  this  proposal  is  to  develop  synthetic 
chromatophores  that  display  color  electrically.  To  create  stretchable  synthetic  chromatophores, 
we  synthesized  high  density  arrays  of  individually  addressable  elastomeric  light  emitting 
capacitors  and  used  a  passive  matrix  addressing  method.  In  this  report,  we  cover  the  results  of 
two  proposal  periods:  September  2015  -  August  2016  and  September  2016  -  July  2017. 

Section  I  (September  2015  -  August  2016): 

Objective 

This  proposal  is  about  enabling  methods  to  better  understand  visual  perception  as  it  pertains  to 
dynamic  camouflage.  Cephalopod’s  (e.g.,  octopuses  and  cuttlefish)  make  themselves  invisible  to 
extremely  sophisticated  visual  predation  using  dynamic  camouflage.  Despite  their  remarkable 
abilities,  these  animals  use  only  three  types  of  coloration  patterns:  Uniform,  Mottled,  and 
Disruptive.  Overlaid  on  this  color  camouflage  are  many  other  disguise  techniques;  the  cuttlefish, 
for  example,  also  uses  pattern  generation,  skin  texture,  and  body  posture  to  disguise  or  reveal 
themselves.  Though  some  basic  explanations  of  how  these  animals  perform  these  maneuvers  and 
when  they  choose  to  implement  them  has  recently  been  explained,  true  understanding  of  how  to 
control  and  use  dynamic  camouflage  in  a  human  context  will  only  be  revealed  using  synthetic 
systems,  instead  of  animal  models.  The  goal  of  this  proposal  is  to  develop  synthetic 
chromatophores  that  display  color  electrically.  To  create  stretchable  synthetic  chromatophores, 
we  synthesized  high  density  arrays  of  individually  addressable  elastomeric  light  emitting 
capacitors  and  used  a  passive  matrix  addressing  method.  Importantly,  we  also  discovered  an 
entirely  new  method  for  achieving  stretchable  displays  based  on  a  new  technology  we 
developed  based  on  elastomeric  optical  waveguides. 


Approach 


•  Synthesize  highly  extensible,  transparent  conductors  and  insulators  that  can  be 
photopatterned  at  high  resolution. 

•  Use  microfabrication  techniques  to  pattern  hyperelastic  light  emitting  capacitors 
(HLECs)  that  increase  in  size  and  emit  light  with  applied  voltage. 

•  Explore  the  trade-offs  between  increasing  HLEC  density  and  maintaining  control 
authority  over  each  synthetic  chromophore. 

•  Explore  the  HLECs  capabilities  as  tactile  sensors.  Application  of  these  skins  to  soft 
machines  will  not  only  allow  them  to  change  color,  but  also  afford  closed-loop  control 
over  posture. 

Relevance  to  Army 

Tough,  stretchable  display  technology  is  important  for  many  reasons:  (i)  by  conformally  wrapping 
around  any  object,  they  can  bring  attention  to  or  hide  sensitive  equipment,  vehicles,  or  personnel; 
(ii)  by  undergoing  large  area  changes,  the  displays  can  be  portable  yet  stretched  into  large  and 
more  visible  displays  or  active  camouflage;  (iii)  it  provides  a  technological  pathway  towards 
understanding  how  cephalopods  use  their  posture,  skin  texture,  and  coloration  for  camouflage  and 
display. 

Accomplishments  for  Reporting  Period 

•  We  reported  the  first  article  on  hyperelastic  light  emitting  capacitors  (HLEC), 
resulting  in  a  display  with  the  largest  reported  strain  to  failure  (~500%  linear 
strain).  The  HLEC  may  actually  fail  at  higher  strains,  the  display  stopped  working  at 
500%  strain  due  to  experimental  limitations.  (Fig.  1-3). 

•  We  reported  the  first  color  changing  soft  robot  that  is  electrically  controlled.  This  soft 
robot  is  composed  of  silicone  elastomer  and  the  display  is  laminated  onto  its  skin,  which 
deforms  by  100’s%  strain.  (Fig.  4). 

•  We  used  the  HLECs  to  not  only  display  color  on  the  soft  robots,  but  also  sense  touch  and 
for  feedback  control  of  the  crawling  soft  robot.  (Fig.  4). 

•  We  developed  a  photolithography  and  transfer  printing  method  to  scale  up  the  pixel 
density  of  the  HLEC  display  and  build  a  passive  matrix  controller  to  display  individual 
pixel  light  emission.  (Fig.  5-7). 

•  The  high  density  HLEC  array  also  serves  as  a  multi-point  touch  sensor.  (Fig.  8). 

•  While  discussing  collaboration  with  Dr.  Roger  Hanlon,  it  became  evident  that  the  ability 
to  control  the  polarization  of  reflected  light  is  important  to  study  cuttlefish  vision  and. 


commensurately,  camouflage.  In  order  to  control  the  polarization  of  our  HLECs,  it  would 
require  a  stretchable  polarizer  layered  onto  the  system;  an  added  complexity  in 
fabrication. 

Collaborations  and  Technology  Transfer 

•  Provisional  patent  applications: 

o  62/250,172:  Stretchable  Electroluminescent  Devices  and  Methods  Making  and 
Using  Same;  Photolithography  procedure  for  mm^  area  HLEC  pixels  patterned 
into  RGB  arrangements  using  transfer  printing 
o  62/317,834:  Ultra-Stretchable  Electroluminescent  Device;  Ionic  hydrogel 

capacitors  with  embedded  light  emitting  semiconducting  particles  in  the  dielectric 
layer. 

•  Portable  system  developed  for  use  in  experiment  with  cuttlefish.  Peele  et  al.,  IEEE  RAM 
(in  preparation).  Samples  sent  to  Dr.  Habtour  for  dynamic  mechanical  analysis,  results  to 
be  included  in  research  paper. 

•  Talk  at  ARE,  invite  by  Dr.  Slipher.  Continuing  collaboration  on  variable  compliance  wings 
and  stretchable  displays. 

•  Established  a  collaboration  with  Dr.  Roger  Hanlon  to  determine  feasibility  of  stretchable 
optical  lightguides  for  studying  vision  in  cuttlefish. 

•  I  Corps  grant  submitted  (08/3 1/2016)  for  integrating  stretchable  displays  into  haptic 
feedback  gloves  for  rehabilitation  and  gaming  using  augmented  and  virtual  reality. 

Resulting  Journal  Publications  During  Reporting  Period 

■  Larson  C,  Peele  B,  Li  S,  Robinson  S,  Totaro  M,  Beccai  L,  Mazzolai  B,  Shepherd  R.  Highly 

stretchable  electroluminescent  skin  for  optical  signaling  and  tactile  sensing.  Science  (2016). 

■  Li  S,  Peele  B,  Larson  C,  Zhao  H,  Shepherd  R.  A  stretchable  multicolor  display  and  touch 

interface  using  photopatterning  and  transfer  printing.  Adv.  Mat.  (2017). 

Graduate  Students  Involved  During  Reporting  Period 

•  Chris  Larson  (PhD  candidate) 

•  Bryan  Peele  (PhD  candidate) 

•  Shuo  Li  (PhD  candidate) 

•  Jason  Cortell  (Staff) 

Awards,  Honors  and  Appointments 

•  PI  Shepherd  selected  to  attend  the  National  Academy  of  Engineering’s  United  States 
Frontiers  of  Engineering  meeting,  2016. 


Section  II  (September  2016  -  July  2017): 


Objective 

This  proposal  is  about  enabling  methods  to  better  understand  visual  perception  as  it  pertains  to 
dynamic  camouflage.  Cephalopod’s  (e.g.,  octopuses  and  cuttlefish)  make  themselves  invisible  to 
extremely  sophisticated  visual  predation  using  dynamic  camouflage.  Despite  their  remarkable 
abilities,  these  animals  use  only  three  types  of  coloration  patterns:  Uniform,  Mottled,  and 
Disruptive.  Overlaid  on  this  color  camouflage  are  many  other  disguise  techniques;  the  cuttlefish, 
for  example,  also  uses  pattern  generation,  skin  texture,  and  body  posture  to  disguise  or  reveal 
themselves.  Though  some  basic  explanations  of  how  these  animals  perform  these  maneuvers  and 
when  they  choose  to  implement  them  has  recently  been  explained,  true  understanding  of  how  to 
control  and  use  dynamic  camouflage  in  a  human  context  will  only  be  revealed  using  synthetic 
systems,  instead  of  animal  models.  The  goal  of  this  proposal  is  to  develop  synthetic 
chromatophores  that  display  color  electrically.  To  create  stretchable  synthetic  chromatophores, 
we  synthesized  high  density  arrays  of  individually  addressable  elastomeric  light  emitting 
capacitors  and  used  a  passive  matrix  addressing  method.  In  this  proposal  period,  we  created  a 
portable  power  system  to  operate  the  synthetic  chromatophores  (called  HLECs),  as  well  as 
use  them  as  touch  sensors.  We  applied  this  system  to  a  spherical  haptic  interface  that 
simulates  the  game  Simon®.  Further,  we  extended  this  concept  to  a  high  density  spherical 
sensor  surface  for  recognizing  gesture  via  convolutional  neural  networks  (CNNs). 

Approach 

•  Mold  a  circular  array  of  HLECs,  each  emitting  light  at  different  frequencies. 

•  Use  small  high  voltage  amplifiers  and  high  voltage  relies  in  a  compact  form,  powered  by 
LiPo  batteries  to  power  the  circular  HLEC  array. 

•  Integrate  an  air  pump  into  the  portable  system  to  actuate  the  haptic  interface. 

•  Feed  the  high  density  sensor  array  into  a  machine  learning  algorithm  to  interpret  haptic 
input  for  computer  control. 

Relevance  to  Army 

Tough,  stretchable  display  technology  is  important  for  many  reasons:  (i)  by  conformally  wrapping 
around  any  object,  they  can  bring  attention  to  or  hide  sensitive  equipment,  vehicles,  or  personnel; 
(ii)  by  undergoing  large  area  changes,  the  displays  can  be  portable  yet  stretched  into  large  and 
more  visible  displays  or  active  camouflage;  (iii)  it  provides  a  technological  pathway  towards 
understanding  how  cephalopods  use  their  posture,  skin  texture,  and  coloration  for  camouflage  and 
display;  (iv)  by  interpreting  their  deformation,  these  skins  can  be  used  for  wearable  sensor 
networks  for  health  monitoring  and  non-verbal  communication. 

Accomplishments  for  Reporting  Period 


•  We  reported  a  paper  on  Untethered  Stretchable  Displays  for  Haptic  Interaction.  This 
display  is  powered  by  batteries,  changes  shape  from  a  flat  disk  to  spherical  orb,  and 
senses  touch  while  emitting  light.  (Fig.  9-14). 

•  We  reported  the  first  use  of  machine  learning  to  interpret  mechanical  deformation  as 
gestures.  We  pressurized  a  5x5  grid  of  stretchable  capacitors  into  a  hemisphere  and  used 
it  as  a  haptic  interface.  By  classifying  different  the  hemisphere  deformations,  we  were 
able  to  train  gestures  to  control  a  game  of  Tetris.  (Fig.  15-18). 

•  Two  students  (Bryan  Peele  &  Chris  Larson)  successfully  defended  their  theses  and 
graduated. 

Collaborations  and  Technology  Transfer 

•  Dr.  Habtour  performed  FEA  to  predict  what  pressures  are  required  to  achieve  particular 
hemispherical  shapes  of  the  inflated  HLEC  array  (paper  in  review). 

•  Talk  at  ARE,  invite  by  Dr.  Walsh,  to  discuss  new  ideas  for  powering  robots  (e.g..  Robot 
Blood  and  Cannibalistic  Skeletal  Frames. 

•  The  discussion  we  started  with  Roger  Hanlon  in  this  grant  resulted  in  a  paper  published 
from  another  grant  on  texture  camouflage  (Science). 

•  We  received  an  I  Corps  grant  to  explore  commercialization  of  this  and  related  technology 
(paper  from  that  grant  submitted  to  IEEE). 

•  Began  a  collaboration  with  NVIDIA  Research  for  using  their  Jetson  micro  AI  computing 
device  for  gesture  interpretation  (paper  in  review). 

Resulting  Journal  Publications  During  Reporting  Period 

■  Peele  B,  Li  S,  Larson  C,  Cortell  J,  Habtour  H,  Shepherd  R.  Untethered  stretchable  displays  for 

tactile  interaction.  Soft  Robotics  (accepted). 

■  Larson  C,  Spjut  J,  Knepper  R,  Shepherd  R.  A  Deformable  Interface  that  Learns  Touch  Gestures 

using  Deep  Neural  Networks.  Published  on  arXiv  (https://arxiv.org/abs/1706.02542I. 

■  Li  S,  Zhao  H,  Shepherd  R.  Flexible  and  stretchable  sensors  for  fluidic  elastomer  actuated  soft 

robots.  MRS  Bulletin  (2017). 

Graduate  Students  Involved  During  Reporting  Period 

•  Chris  Larson  (PhD  candidate) 

•  Bryan  Peele  (PhD  candidate) 

•  Shuo  Li  (PhD  candidate) 

Awards,  Honors  and  Appointments 


PI  Shepherd  received  ONR  Young  Investigator  Award  during  this  time  period. 
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Fig.  1. 

Hyperelastic  light  emitting  capacitor  (HLEC).  (A)  Image  of  the  HLEC  conforming  to  the  end  of  a  pencil. 
(B)  Exploded  view  of  the  HLEC  showing  its  five-layer  structure  consisting  of  a  -1  mm  thick 
electroluminescent  layer  (ZnS-Ecoflex  00-30)  that  is  sandwiched  between  two  PAM-LiCl  hydrogel  electrodes 
and  encapsulated  in  Ecoflex  00-30.  (C)  Stress-stretch  curves  of  Ecoflex  00-30,  the  electroluminescent  layer, 
the  hydrogel,  and  the  composite  device. 
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Fig.  2.  The  capacitive  and  luminescent  behavior  of  the  HLEC  display  under  uniaxial  stretching.  (A)  A 

nominal  electric  field  of  -25  kV  cm'^  was  applied  to  the  HLEC  at  the  start  of  the  uniaxial  test.  Five  lengths 
were  measured  using  Image- J  to  obtain  Ai  across  the  width  of  the  illuminated  portion  of  the  tensile  bar.  We 
report  the  mean  and  standard  deviation  of  those  measurements.  At  an  engineering  strain  (grip-to-grip)  of 
395%,  we  measured  the  mean  strain  of  the  illuminated  portion  to  be  487%,  with  a  range  of  420  -  549%.  (B) 
The  capacitance  of  the  HLEC  as  a  function  of  its  uniaxial  stretch  (n  =  4).  (C)  The  relative  illuminance  of  the 
HLEC  versus  its  uniaxial  stretch  (n  =  4)  plotted  alongside  predicted  values  (supplementarv  online  text). 
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Fig.  3.  Multi-pixel  electroluminescent  displays  fabricated  via  replica  molding.  The  device  measures  5  mm 
thick,  with  each  of  the  64  pixels  measuring  4  mm.  We  show  the  devices  in  various  states  of  deformation  and 
illumination:  (A)  undeformed,  (B)  stretched,  (C)  conformed  to  a  finger,  (D)  folded,  (E)  rolled,  (F-H)  subsets 
of  pixels  activated,  and  (I-J)  subsets  of  pixels  activated  while  being  deformed. 


Fig.  4.  HLEC  skins  endow  soft  robots  with  the  ability  to  sense  their  actuated  state,  environment,  and 
communicate  optically.  (A)  Schematic  of  a  three-chambered  fluidically  actuated  crawler.  A  series  of  three 
independently  actuated  pneumatic  chambers  are  embedded  between  the  HLEC  skin  (top)  and  a  strain  limiting 
layer  (bottom).  (B)  Capacitance  plotted  versus  the  actuation  amplitude,  which  we  define  as  the  relative  change 
in  deflection  between  the  uninflated  and  fully  inflated  states  {n  =  5).  (C)  A  firm  finger  press  induces  a  -25% 
increase  in  capacitance.  (D)  Change  in  capacitance  versus  applied  pressure.  We  observe  a  negligible  change  in 
the  capacitive  response  of  the  sensors  over  a  period  of  120  hours.  (E)  Array  of  three  HLEC  panels,  each 
emitting  a  different  wavelength  through  selective  doping  of  the  EL  phosphor  layer.  Each  HLEC  panel  is 
activated  independently.  (F)  An  undulating  gait  is  produced  by  pressurizing  the  chambers  in  sequence  along 
the  length  of  the  crawler.  As  each  pneumatic  chamber  is  pressurized,  the  outer  electroluminescent  skin  is 
stretched,  increasing  the  electric  field  across  the  EL  layer  and  thus  the  luminescence. 
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Fig.  5.  (a)  Schematic  illustration  of  the  process  flow  for  photopatterning  and  transfer  printing  to  form 
the  multicolor  electroluminescent  dielectric  layer,  (b)  Exploded  view  of  the  m-HLEC  showing  its  five 
layers  comprising  a  -1.3  mm  thick  EL  dielectric  layer  (ZnS-silicone)  that  is  sandwiched  between  top  and 
bottom  hydrogel  electrodes  in  orthogonal  layouts  and  protected  by  a  silicone  encapsulation  layer. 


Fig.  6.  (a)  Stress-strain  curve  of  the  ZnS-silicone  gel-silicone  elastomer  composite  light-emitting 
dielectric  layer.  The  nominal  stress,  a,  is  defined  as  the  force  applied  on  the  dog-bone  shaped  composite, 
divided  by  the  cross-sectional  area  of  the  undeformed  middle  section,  (b)  Luminance  of  m-HLEC  fabricated 
using  three  different  colored  EL  phosphor  powders  in  response  to  area  strains,  (c)  Cross-sectional  scanning 
electron  microscopy  (SEM)  image  showing  the  distribution  and  morphology  of  EL  particles  embedded  in 
silicone  gel  matrix  and  encased  in  silicone  elastomer  encapsulation  for  an  unstrained  sample,  (d)  Cross- 
sectional  SEM  image  for  a  uniaxially  strained  sample.  White  dotted  circles  represent  the  locations  of  EL 
particles  in  the  unstrained  state. 


a 

.  i- 

'  ^  m* 

2  cm 

/ 

C 

t 

'  c 

d 

% 

V' 

e 

\ 

■  y'  /  ^ 

f 

Fig.  7:  m-HLEC  showing  dynamic  patterns  under  various  deformations,  (a)  Illuminating  all  pixels, 
undeformed;  (b)  illuminating  left  and  right  four  columns,  stretching;  (c)  illuminating  top  right  comer  5x5 
pixels,  wrapping  around  a  finger;  (d)  illuminating  all  pixels,  folding;  (e)  illuminating  middle  six  columns, 
rolling;  (f)  illuminating  center  4x4  pixels,  twisting. 
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Fig.  8:  Touch  sensing  capability  of  the  m-HLEC.  (a)  Normalized  relative  capacitance  in  response  to  applied 
force  at  four  randomly  selected  pixels  in  the  range  of  0-4.5  N,  for  both  unstrained  and  strained  (£area=100%) 
states,  (b)  Greyscale  gradient  in  the  topographical  map  demonstrates  normalized  relative  capacitance  in 
response  to  applied  force  at  unstrained  and  strained  (£area=100%  and  200%)  states,  (c)  Pixels  shown  in  black 
have  been  identified  as  being  pressed  using  a  weighted  filter  that  compares  the  normalized  relative  capacitance 
of  each  pixel  to  that  of  its  neighbors. 


Figures  (Section  II) 


Fig.  9.  Four  hyperelastic  light-emitting  capacitors  (HLECs)  form  an  untethered  soft  interface  capable  of 
emitting  light  and  sensing  touch.  (A)  The  membrane  is  pressurized  using 

an  integrated  pump.  (B)  Fully  inflated  HLEC  interface  with  one  panel  illuminated  using  applied  voltage,  F-4 
kV,  and  frequency,/ -  1  kHz. 
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Fig.  10.  Stretch  of  the  elastomeric  membrane  during  inflation.  (A)  Aggregate  stretch  across  the  profile  of 
the  HLEC  membrane  as  internal  pressure  is  applied.  Experimental  results  are  compared  to  results  of  a 
numerical  simulation.  Profiles  of  the  device  are  shown  alongside  vertical  displacement,  u,  for  pressures  of  1.4 
kPa  (B),  2.8  kPa  (C),  4.1  kPa  (D)  and  5.5  kPa  (E).  Simulations  performed  by  Dr.  Habtour  (ARE). 
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Fig.  12.  Touch  sensing  capabilities  of  the  HLEC  panels.  (A)  Sequence  of  touches  to  the  HLEC  panels, 
starting  with  the  white  panel  in  the  bottom  left  and  proceeding  clockwise  around  the  device.  A  sequence  of 
twelve  touches  (three  per  panel)  is  repeated  for  the  unpressurized  and  pressurized  states.  (B)  Representative 
touch  on  the  blue  panel  in  the  unpressurized  state.  (C)  Representative  touch  on  the  blue  panel  in  the 
pressurized  state.  (D)  Relative  capacitance  for  each  HLEC  panel  as  they  are  pressed  in  the  unpressurized  state. 
(E)  Relative  capacitance  for  each  HLEC  panel  as  they  are  pressed  in  the  pressurized  state.  The  dashed  grey 
lines  in  (D)  and  (E)  indicate  the  threshold  used  to  indicate  whether  a  panel  has  been  pressed.  The  colored 
rectangles  below  the  capacitance  data  represent  which  panel,  if  any,  have  been  identified  as  being  pressed. 


0  2  4 

Inflation  Pressure,  AP  (kPa) 


Fig.  13.  Change  in  luminance  during  deformation.  (A)  Luminance  for  the  blue  panel  as  the  membrane  is 
pressurized.  Error  bars  represent  the  standard  error  calculated  using  three  measurements.  (B-C)  Pressurized 
HLEC  panel  being  deformed  with  a  finger  press  while  illuminated.  The  images  on  the  right  show  the  relative 
luminance  of  the  device,  with  yellow  coloration  corresponding  to  the  highest  luminance. 


Fig.  14.  Illumination  of  the  HLEC  panels.  Blue  (A),  orange  (B),  green  (C)  and  white  (D)  panels  are 
illuminated  while  the  membrane  is  uninflated.  Blue  (E),  orange  (F),  green  (G)  and  white  (H)  panels  are 
illuminated  while  the  membrane  is  pressurized  to  ~4.1  kPa. 
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Fig.  15.  Photographs  of  the  OrbTouch  device,  (a)  Its  embedded  capacitors  capture  shape  changes  caused  by 
human  touch,  (b)  The  internal  components  of  OrbTouch  consist  of  an  embedded  RBPI3  computer,  ADC,  and 
air  compressor  used  to  control  pressure  in  the  orb. 
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Fig.  16.  Capacitance  measurement  method,  (a)  To  measure  capacitance,  we  set  one  vertical  electrode  HIGH 
(+3.3  VDC)  and  monitor  the  induced  voltages  on  the  orthogonal  electrodes  using  an  ADC,  which  relays  the 
signals  to  the  RBPI3  over  SPI  serial.  During  each  measurement,  there  is  one  pin  set  HIGH,  and  one  pin  that  is 
read;  the  remaining  eight  electrodes  are  connected  to  ground  to  minimize  cross  talk  between  neighboring 
electrodes  and  electromagnetic  interference,  (b)  Equivalent  measurement  circuit.  The  i,j^^  capacitor  is 
represented  by  Cij.  The  nominal  capacitance  of  our  sensors  is  41.2  pF  (SD  =  2.9  pF).  We  use  a  Rm  =  50 
Mohm  inline  resister  to  yield  an  RC  time  constant  of  T^2  ms.  We  use  a  second  capacitor.  Cm  =  1  pF,  to  flip  the 
polarity  of  the  measured  Vm  voltages. 
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Fig.  17.  Computational  graph  of  the  inference  (CNN-3D)  and  regression  (CNN-2D)  models.  Both 
networks  have  two  hidden  convolutional  layers  and  two  hidden  fully  connected  layer.  The  kernel  size,  k,  and 
stride,  s,  of  each  convolutional  operation  are  provided.  Network  CNN-3D  accepts  as  input  a  sliding  window  of 
m  discrete  sensor  readings  (5x5xm)  and  outputs  a  probability  distribution  over  nc  classes  using  a  softmax 
activation  on  the  output.  Because  the  information  in  a  gesture  is  inherently  spatiotemporal,  we  convolve  a  3D 
kernel  over  both  the  spatial  and  temporal  dimensions  of  the  input  to  capture  relevant  features.  Network  CNN- 
2D  accepts  a  5x5  sensor  matrix  and  outputs  a  continuous  valued  vector  using  a  tanh  activation  on  the  output 

layer. 
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Figure  18.  Application  of  OrbTouch  to  the  popular  game  Tetris,  (a)  Photograph  of  OrbTouch  being  used 
to  control  an  adaptation  of  the  game  Tetris,  (b)  Finger  pressing,  or  poking  are  used  to  translate  the  Tetromino 
left,  right,  and  down  (L,R,D).  (c)  Pinching  is  used  to  drop  the  Tetromino  directly  to  the  bottom  of  the  grid.(d) 
Clockwise  rotation,  or  twisting,  is  used  to  rotate  the  Tetromino  90  deg  in  the  clockwise  direction,  (e) 
Counterclockwise  rotation  is  used  to  rotate  the  Tetromino  90  deg  in  the  counterclockwise  direction,  (f) 
OrbTouch  software  diagram.  The  first  processing  step  executes  capacitance  measurements,  CNN-2D,  and 
CNN-3D,  while  the  second  step  generates  a  command  and  updates  the  model  inputs  for  the  next  timestep. 
Each  of  these  steps  is  multithreaded.  We  use  debouncing  filter  prior  to  sending  commands  to  the  host  (via 
Bluetooth).  Each  cycle  of  compute  takes  ~86  ms,  which  fits  within  our  100  ms  target. 


